Repository navigation
Conversation
沿用 DeepSpeed 样例的双机看护结构,把四个 workflow 拆为 qs-a2 / qs-a3 / merge 三个 job,qs-a3 使用 A3 runner 与 910C 镜像,并按现有 A3 job 的约定 挂载 /data/ci-cache 缓存。 A3 资源池串行调度,四条流水线与 DeepSpeed 按 2 小时窗口错峰。A3 runner 为 双卡,把写死卡数的断言放宽为 fuzzy,并为 comfyui、minionerec 补上 ASCEND_RT_VISIBLE_DEVICES=0 以保持单卡执行。
Contributor
|
📖 文档预览:https://ascend.github.io/docs/pr-preview/pr-277/ 本 PR 改动了多个项目: 构建自 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
沿用 DeepSpeed 样例(#273)引入的双机看护结构,为 evalscope、modelscope、comfyui、minionerec 四个组件补齐 A3 文档看护。每个 workflow 由单个 job 拆为
qs-a2、qs-a3两个执行 job 与一个mergejob,两者调用同一份quick-start-template.yml,结果由merge-quick-start-results合并为单个 artifact。qs-a3使用 runnerlinux-aarch64-a3-800i-ascend-docs-fed-2与镜像cann:9.1.0-a3-ubuntu22.04-py3.12,其余输入(上游仓库、受监控文档、测试入口、超时)与qs-a2一致。A3 资源池当前串行调度,因此四条流水线与 DeepSpeed 按 2 小时窗口错峰(UTC):DeepSpeed 00:00、evalscope 03:00、modelscope 05:00、comfyui 07:00、minionerec 09:00,各窗口 12 小时后重复。
A3 runner 为双卡,
check-torch中固定为count: 1的断言在双卡环境下必然失败,已改为fuzzy='xxx'与count: xxx;同时为 comfyui、minionerec 的prepare_environment补充ASCEND_RT_VISIBLE_DEVICES=0,与 evalscope、modelscope 的既有做法保持一致,确保测试仍按单卡执行。evalscope 在 A3 上输出的模型名将点号渲染为下划线(
Qwen2_5-0_5B-Instruct),run-eval的断言由匹配模型名改为匹配评测完成行Finished evaluation for ... on ['gsm8k', 'arc'],两种平台的输出均可命中,数据集缺失时仍会失败。qs-a3按现有 A3 job 的约定挂载模型与依赖缓存(/data/ci-cache下按组件分目录),避免首次运行重复下载模型权重与数据集;qs-a2沿用 runner 侧已有的/root/.cache,不额外挂载。四份文档的硬件说明与镜像表补充 A3(
Atlas 900 A3/Ascend 910C、cann:9.1.0-a3-ubuntu22.04-py3.12)。验证方式:四个 workflow 分别 dispatch 一次,确认
mergeartifact 中qs-a2与qs-a3两份 result.json 的 conclusion 均为 passed。